A data transformation system takes one or more datasets as input and applies business logic to the input datasets to generate one or more output data sets. Data may be first ingested as a raw data from a data source. Ingestion of data into a data transformation system can be challenging, as the ingested data may not be formatted as an expected dataset. For example, the ingested data may be formatted as COBOL data, XML data, or some other format, and the data transformation system expects datasets to be in a CSV format. Furthermore, ingested data may not have a schema definition. Additionally, ingested data may not have a configuration for access control list (ACL) permission settings to protect the data and limit that users and/or accounts have access to the ingested data. Thus, what is needed is a system to automate the ingestion of data into a data transformation system.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
While each of the figures illustrates a particular embodiment for purposes of illustrating a clear example, other embodiments may omit, add to, reorder, and/or modify any of the elements shown in the figures.